Skip to content

feat: add resource dataproc jobs controller #4231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

xiaoweim
Copy link
Collaborator

@xiaoweim xiaoweim commented Apr 2, 2025

  • Run make ready-pr to ensure this PR is ready for review.
  • Perform necessary E2E testing for changed resources.

@xiaoweim xiaoweim force-pushed the resource-dataproc-jobs branch from 91d6f4e to 4fd51ce Compare April 2, 2025 20:33
@xiaoweim xiaoweim changed the title [WIP] feat: add resource dataproc jobs controller feat: add resource dataproc jobs controller Apr 2, 2025
@xiaoweim xiaoweim force-pushed the resource-dataproc-jobs branch 3 times, most recently from f595c8a to f64b6a6 Compare April 3, 2025 00:50
@xiaoweim xiaoweim force-pushed the resource-dataproc-jobs branch 5 times, most recently from 5e20145 to 3ab15cb Compare April 4, 2025 01:06
@yuwenma
Copy link
Collaborator

yuwenma commented Apr 7, 2025

Thanks! Could you share the new git-diff between real and mock?

@xiaoweim
Copy link
Collaborator Author

xiaoweim commented Apr 7, 2025

Thanks! Could you share the new git-diff between real and mock?

Sure, I can record it again.

@xiaoweim xiaoweim force-pushed the resource-dataproc-jobs branch from 3ab15cb to b6bc8c3 Compare April 7, 2025 19:11
Copy link
Collaborator

@yuwenma yuwenma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @jingyih (a service-generated-id)

Is the controller diff (compare to scifi template) from LLM?

@xiaoweim
Copy link
Collaborator Author

/cc @jingyih (a service-generated-id)

Is the controller diff (compare to scifi template) from LLM?

If you referring to the changes in controller to add the server-generated id, I have added that manually.

@xiaoweim
Copy link
Collaborator Author

/assign @jingyih

Could you please take a look at this PR?

@@ -34,6 +34,21 @@ func redisClusterFuzzer() fuzztesting.KRMFuzzer {
)

f.UnimplementedFields.Insert(".name") // Identifier
f.UnimplementedFields.Insert(".backup_collection")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These "UnimplementedFields" were already added in this file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the go dependency was updated and the updated proto has these fields but KCC currently do not support.

@xiaoweim xiaoweim force-pushed the resource-dataproc-jobs branch from 84f4efd to ffb8e28 Compare May 30, 2025 21:56
Copy link
Collaborator

@jingyih jingyih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

It looks like we are not testing the patch RPC?

Not a blocker for alpha resource. We can add it later.

Labels: desiredProto.Labels,
}
// Populate the job type specific field
if a.desired.Spec.HadoopJob != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit confused. These conversion should be handled by mapping functions? I would expect DataprocJobSpec_ToProto to handle this.

@google-oss-prow google-oss-prow bot added the lgtm label May 31, 2025
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jingyih

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit de268fd into GoogleCloudPlatform:master May 31, 2025
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants